Analysis and Modeling of MapReduce’s Performance on Hadoop YARN

With the rapid growth of technology, scientists have realized the challenge of efficiently analyzing large data sets since the beginning of 21 century. Increases in data volume and data complexity shift scientists’ focus to parallel, distributed algorithms running on clusters. In 2004, Jeffrey Dean and Sanjay Ghemawat from Google introduced a new programming model to store and process large data sets, called MapReduce[2]. Apache Hadoop, an opensource software framework, which uses MapReduce as its data-processing layer, was developed at Yahoo as early as 2006 and evolved to a stable platform by 2011. Although Hadoop has been widely used in industry, its performance characteristics are not well understood. This paper, following Hadoop’s workflow, analyzes the factors that influence the running time of each phase in Hadoop execution. Given those factors, our goal is to model the performance of MapReduce applications.